CLAIMS 



\l. One or more computer readable media having stored thereon a 
plurality of instructions that, when executed by one or more processors, causes the 
one or more processors to perform acts including: 

generatmg a plurality of query vectors by extracting, for each query vector, 
one of a plurality\pf low-level features from the initial image selection; 

selecting a\set of potentially relevant images based at least in part on 
distances between tnfe plurality of query vectors and a plurality of feature vectors 
corresponding to low-level features of a plurality of images; 

receiving feedback, regarding the relevance of one or more images of the set 
of potentially relevant imagfes; 

generating a new plurality of query vectors based at least in part on the 
feedback; \ 

generating a weighting of feature elements based at least in part on the 
feedback; and \ 

selecting a new set of potentially relevant images based at least in part on 
both the weighting of feature element^ and distances between the new plurality of 
query vectors and the plurality of feaUnA vectors. 
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One or more computer readable media as recited in claim 1, wherein 
the selecting a new set of potentially relevant images comprises using a matrix in 
determiningxthe distance between one of the new plurality of query vectors and 
one of the plurality of feature vectors, and further comprising dynamically 
selecting the matrix based on both a number of images in the set of potentially 
relevant images for which relevance feedback was input and a number of feature 
elements in the one feature vector. 



3. One or more computer readable media as recited in claim 2, wherein 
the dynamically selecting comprises using a diagonal matrix if the number of 
images in the set of potentially relevant images for which relevance feedback was 
input is less than the number of feature elements in the one feature vector, and 
otherwise using a full matrix. 



4. One or more computer readable^media as recited in claim 2, wherein 
the dynamically selecting comprises: 

if the number of images in the set of potentially relevant images for which 
relevance feedback was input is not less than the number of feature elements in the 
one feature vector, then using one matrix that transforms the query vector and the 
one feature vector to a higher-level feature space anoythen using another matrix 
that assigns a weight to each element of the transformed query vector and the 
transformed feature vector; and 

if the number of images in the set of potentially relevant images is less than 
the number of feature elements in the one feature vector, then using a matrix that 
assigns a weight to each element of the query vector and the onevfeature vector. 
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5. \ One or more computer readable media as recited in claim 2, wherein 
X represents\an image matrix that is generated by stacking AT feature vectors 
corresponding to\the set of potentially relevant images for which relevance 
feedback was received and resulting in an (N x K) matrix, C represents a 
weighted covariance matrix of X , det( C ) represents the matrix determinant of C , 
and the matrix comprises a Ml matrix ( W * ) that is generated as follows: 

*F* = (det(C))^C _1 . \ 

6. One or more computer readable media as recited in claim 2, wherein 
represents the kk ih element of matrix x k represents the A; 1 * 1 feature element, 

a k represents the standard deviation of the sequence of x k *s, the matrix comprises 
a diagonal matrix with each diagonal element ( w M \being generated as follows: 

1 \ 
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7. \pne or more computer readable media as recited in claim 1 , wherein 
N represents the number of images in the set of potentially relevant images for 
which relevance feedback has been received, n n represents the relevance of image 
n in the set of images, k represents a transposition of a vector generated by 
concatenating the indivi&ial n n values, and X represents an image matrix that is 
generated by stacking N training vectors corresponding to the set of potentially 
relevant images into a matrix, and wherein each new query vector ( q ) of the new 
plurality of query vectors is generated as follows: 

"> K X \ 

8. One or more computer readable mUa as recited in claim 1, wherein 
f i represents a summation, over the images in\ the set of potentially relevant 
images, of a product of a relevance of the image ana a distance between the query 

vector and the feature vector, and wherein the selecting a new set of potentially 

* \ 
relevant images comprises combining, for each image, a weighted distance 

between the plurality of query vectors and the plurality of feature vectors, and 

wherein the weight (u^ for each of a plurality (/) of distances between a query 

vector and a corresponding feature vector is calculated as: \ 

9. One or more computer readable media as recited in claim 1, wherein 
the receiving feedback comprises receiving feedback from a user A 
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10. One or more computer readable media as recited in claim 1, wherein 
the low-leVel features include: a color moments feature, a wavelet based texture 
feature, and V water- fill edge feature. 

11. A method of selecting between two types of matrixes to be used to 
weight, based on relevance feedback, a plurality of feature elements for image 
retrieval, the method comprising: 

selecting one of\the two types of matrixes based on both a number of 
previously retrieved relevant images and a length of a feature vector including the 
plurality of feature elements\ 

12. A method as recited in claim 11, wherein the selecting comprises 
selecting one of the two types of matrixes based on both a number of previously 
retrieved potentially relevant images, which were identified by a user as being 
relevant, and the length of the featum vector including the plurality of feature 
elements. \ 

13. A method as recited in claim Yl, wherein the plurality of feature 
elements are all elements of the same feature. \ 

14. A method as recited in claim 11, wnerein the selecting comprises 
using a first type of matrix if the number of retrieveckrelevant images is less than 
the length of the feature vector, and otherwise using a second type of matrix. 
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15. \ A method as recited in claim 14, wherein the first type of matrix 
comprises a diagonal matrix and wherein the second type of matrix comprises a 
full matrix. \ 

16. A method as recited in claim 11, wherein the selecting comprises 
using a first type of matrixsif the length of the feature vector exceeds the number 
of retrieved relevant images bV at least a threshold amount, and otherwise using a 
second type of matrix. \ 

17. A method as recited imclaim 16, wherein the first type of matrix 
comprises a full matrix and the second type of matrix comprises a diagonal matrix. 

18. One or more computer readable media including a computer 
program that is executable by a processor to peiform the method recited in claim 
11. \ 

19. One or more computer readable media having stored thereon a 
plurality of instructions that, when executed by one or more processors, causes the 
one or more processors to perform acts including: \ 

comparing, for each of a plurality of images, a plurality of feature elements 
from a query vector to a plurality of feature elements from a feature vector 
corresponding to the image; \ 

identifying a number of potentially relevant images based on the 
comparing; \ 
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receiving user feedback regarding the relevancy of one or more of the 
potentially relevant images; 

re-comparing, for each of the plurality of images, the plurality of feature 
elements from tke query vector to the plurality of feature elements from the 
feature vector, including using a matrix to compare the feature elements and 
dynamically selecting^ type of matrix to use based on both the user feedback and 
the number of the plurality of feature elements; 

identifying a new set of potentially relevant images based on the re- 
comparing; and \ 

presenting the new set of potentially relevant images to the user. 

20. One or more computer readable media as recited in claim 19, 
wherein the re-comparing comprises dynamically selecting the type of matrix to 
use based on both a number of the potentially relevant images for which user 
feedback has been received and the numbet of the plurality of feature elements. 

21. One or more computer readable media as recited in claim 19, 
wherein the dynamically weighting comprises Vsing a first type of matrix if the 
number of retrieved relevant images is less thanuhe length of the feature vector, 
and otherwise using a second type of matrix. \ 

22. One or more computer readable media as recited in 21, wherein the 
first type of matrix comprises a diagonal matrix and the second type of matrix 
comprises a full matrix. \ 
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23. \ A method comprising: 
generating a query vector corresponding to a feature of one image; 
identifying a feature vector corresponding to the feature of another image; 
identifyingNa number of training samples for which relevance feedback has 

been received; \ 

if the number Vf training samples either equals or exceeds a threshold 
amount, then determining a distance between the query vector and the feature 
vector including transforming the query vector and the feature vector to a higher- 
level feature space and then assigning a weight to each element of the transformed 
query vector and the transformed feature vector; and 

if the number of trainings samples does not exceed the threshold amount, 
then determining the distance beWeen the query vector and the feature vector 
including assigning a weight to each element of the query vector and the feature 
vector. \ 

24. A method as recited in claim >23, wherein the feature vector includes 
a plurality of feature elements and wherein Vhe threshold amount comprises the 
number of feature elements in the feature vectoV 

25. A method as recited in claim 23, wherein if the number of training 
samples either equals or exceeds the threshold amount, then determining the 
distance (g), where P is a mapping matrix, q is theWery vector, x is the feature 
vector, and A is a weighting matrix, as: \ 

g = (P(q-x)) T A(P(q-x)). \ 
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26. \^ A method as recited in claim 23, wherein if the number of training 

samples does not exceed the threshold amount, then determining the distance (g), 

~* \ ~~ * 
where q is the^ query vector, x is the feature vector, and A is a weighting matrix, 

as: 

S = (?-*)\A(9-*)- 



27. A method as recited in claim 23, further comprising: 

repeating the generating, identifying of the feature vector, identifying of the 

number of training samples, and the determining for each of a plurality of features; 

and 

identifying how closely the image and the other image match each other by 
combining the distances between the query vectors and the feature vectors for the 
plurality of features. 



28. A method as recited m claim 27, wherein the identifying comprises 
calculating a weighted summation of \ach of the individual distances for each of 
the plurality of features. 

29. One or more computer readable media including a computer 
program that is executable by a processor to perform the method recited in claim 
23. 

30. A system comprising: 

a query vector generator to generate a qi\ery vector corresponding to a 
feature of one image; 
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a comparator, coupled to the query vector generator, to, 

identify a feature vector corresponding to the feature of another 

image, 

identify a number of training samples for which relevance feedback 
has been received, 

if the number of training samples either equals or exceeds a 
threshold amount, tWen to determine a distance between the query vector 
and the feature vector including transforming the query vector and the 
feature vector to a higher-level feature space and then assigning a weight to 
each element of the transformed query vector and the transformed feature 
vector, and 

if the number of training samples does not exceed the threshold 
amount, then to determine the distance between the query vector and the 
feature vector including assigning a weight to each element of the query 
vector and the feature vector. 

L A method comprising: 
for oije of a plurality of images and each of a plurality of features, 

^nerating, based on the set of search criteria, a query vector for the 

feature, 

idefeti^Qp^ a feature vector, corresponding to the image, for the 
feature, and 

determinin&how closely the feature vector matches the query vector; 



25 



Lee & Hayes, PLLC 



MS1-610.PA T.APP.DOC 



determining how closely the image matches the set of search criteria based 
on how cldKejjy^ for the plurality of features, the feature vectors match the query 
vectors. ^ 



32.\ A method as recited in claim 31, wherein generating the query 
vector comprises generating the query vector based at least in part on user 
relevance feedback regarding how relevant images previously displayed to a user 



were. 



\ 



33. A method v as recited in claim 31, wherein identifying the feature 
vector comprises: 

identifying a low-levelYeature vector corresponding to the feature; and 
mapping the low-level feature vector to a higher level feature space. 

34. A method as recited in claim 33, wherein the identifying the feature 
vector further comprises incorporating, into the mapping, relevance feedback. 



35. A method as recited in claim 3 1\ wherein the initial search criteria 
comprises an image. 
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3^ A method as recited in claim 31, wherein the determining how 
closely th& feature vector matches the query vector comprises determining a 
distance between the feature vector and the query vector, and wherein the 
determining hVvv closely the image matches the set of search criteria comprises 
calculating a weighted summation of each of the individual distances between the 
feature vectors andihe query vectors. 

37. A method as recited in claim 36, wherein the calculating a weighted 
summation comprises calculating the weighted summation based at least in part on 
user relevance feedback regarding how relevant images previously displayed to a 
user were. 

38. One or more computer readable media including a computer 
program that is executable by a processor to perform the method recited in claim 
31. 



$9. One or more computer readable media having stored thereon a 
plurality of instructions that, when executed by one or more processors, causes the 
one or more processors to perform acts including: 

identifyingSa plurality of query vectors for one image, each query vector 
corresponding to onelt^f p plurality of features; 

identifying a p\jir)^k£ of feature vectors for the other image, each feature 
vector corresponding to one^ the plurality of features; 

for each feature, determining a distance between the corresponding query 
vector and the corresponding feature vector; and 
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combiniiWhe distances to generate a value representing an overall distance 
between the two images. 



40\ One or more computer readable media as recited in claim 39, 
wherein the\identifying the plurality of query vectors comprises extracting the 
plurality of queW vectors from the image. 

41. One or more computer readable media as recited in claim 39, 
wherein the identifying the plurality of query vectors comprises generating the 
plurality of query vectors based at least in part on user relevance feedback 
regarding how relevant images previously displayed to a user were. 

42. One or more conrouter readable media as recited in claim 39, 
wherein the determining the distance between the corresponding query vector and 
the corresponding feature vector includes incorporating, into the determining, user 
relevance feedback regarding how relevant images previously displayed to a user 
were. \ 

43. One or more computer readable media as recited in claim 39, 
wherein the combining the distances comprises calculating a weighted summation 
of each of the individual distances between the feature vectors and the query 
vectors. \ 
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44. One or more computer readable media as recited in claim 43, 
wherein the calcinating a weighted summation comprises calculating the weighted 
summation based at least in part on user relevance feedback regarding how 
relevant images previously displayed to a user were. 

A system comprising: 
a quVy vector generator to identify a query vector, for each of a plurality of 
features, corresponding to one image; 

a comparator, coupled to the query vector generator, to, 

identiiV, /ffcir each of a plurality of features of another image, a 
feature vector fdfc the feature, 

determineV^sjr each of the plurality of features, how closely the 
feature vector matche^he query vector, and 

determine how closely the image matches the other based on how 
closely, for the plurality of\features, the feature vectors match the query 
vectors. 

46. AS^nethod of generating a query vector to compare to a feature 
vector of another im^ee, the method comprising: 

receiving feedba^ regarding the relevance of each image of a set of 
images; 

wherein TV represents the^humber of images in the set of images for which 
user relevance feedback has been rec^ved, n n represents the relevance of image n 
in the set of images, n represents transposition of a vector generated by 
concatenating the individual n n values, anck X represents an image matrix that is 



Lee & Hayes, PLLC 



38 



MSl-6I0.PAT.APP.DOC 



1 

2 
3 
4 
5 
6 
7 
8 
9 
10 
11 
12 
13 
14 
15 
16 
17 
18 
19 
20 
21 
22 
23 
24 
25 



generated\by stacking AT training vectors corresponding to the set of images into a 
matrix; anc 

generating a query vector (q) corresponding to one of a plurality of 
features as follows: 



iV 

2X 



n=\ 



47. One or more computer readable media including a computer 



A 



program that is executable by^a processor to perform the method recited in claim 
46. 



48. A method of generating a weight to apply to distances between 
query vectors and feature vectors wn^n combining the distances, the method 
comprising: 

receiving feedback regarding the relevance of each image of a set of 
images; 

wherein f. represents a summation, over^he images in the set of images, of 
a product of a relevance of the image and a distance between the query vector and 
the feature vector; and 

generating a weight (w, ) for each of a pluralitV (/) of distances between a 
query vector corresponding to one of a plurality (/)\of features and a feature 
vector corresponding to the one of the plurality ( / ) of features as: 

7=1 V J I 
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49. \One or more computer readable media including a computer 
program that A executable by a processor to perform the method recited in claim 
48. 
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50. A systei\comprising: 
a client device; 

a collection of a plurality of images; 
an image server, coupled to the client device and the collection of a 
plurality of images, the image i^erver to receive image retrieval requests from the 
client device and to 3 

receive an initial imagfe selection from the client device, 
generate a plurality of query vectors by extracting, for each query 
vector, one of a plurality of lojv-level features from the initial image 
selection, 

select a set of potentially releVant images based at least in part on 
distances between the plurality of queW vectors and a plurality of feature 
vectors corresponding to low-level features of a plurality of images, 

receive feedback regarding the relevance of one or more images of 
the set of potentially relevant images, 

generate a new plurality of query vectdp based at least in part on the 
feedback, 

generate a weighting of feature elements teased at least in part on the 
feedback, and 
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selectNa new set of potentially relevant images based at least in part 
on both the weighting of feature elements and distances between the new 
plurality of queW vectors and the plurality of feature vectors. 
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